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Abstract 

The exploration and exploitation of space 
will require an optimal combination of human 
beings and machines. The latter may be either on- 
site, or a long distance from the space-based activity. 
Sending sensory input to, and receiving direction 
from, human beings at a remote location is known as 
telepresence and teleoperation. Its uses will include 
mining and processing lunar materials, assembling 
large space structures, and assisting in remote 
emergency surgery. Extraterrestrial teleoperation is 
constrained by the velocity of light, whose round- 
trip travel time between the Earth and the Moon is 
2.6 seconds. The cutting edge in teleoperated 
terrestrial surgery is described. Visual feedback in 
teleoperation is limited by latency, computational 
and bandwidth requirements, image generation lag, 
operator fatigue, etc. Sound feedback can overcome 
some of these limitations. Teleoperational methods 
being developed for terrestrial surgery can be 
extended to various space-based activities. To 
handle these different tasks, various teleoperation 
strategies are described. Serving in place of 
gravitational cues, perhaps virtual audio can help us 
navigate on the high frontier. 

Introduction 

One of the key issues in the planning of a 
space-based operation is the appropriate use of 
human beings and machines. Although many tasks 
in space will be fully automated, many others will 
require a human presence. The roles of human 
beings and machines are usually devised to balance 
several needs, such as cost constraints, the dangers of 
various space environments, and the usefulness of 
human perception, decision-making, and 
adaptability. In situations where human skills are 
needed, but human vulnerability or cost constraints 
prohibit a direct human presence, it may be possible 
for a human being to project his/her skills into a 
distant environment through a radio link. This will 
involve the use of machines sending sensory input 
to, and receiving direction from, human beings at a 
remote location. This is referred to as telepresence 
and teleoperation. Teleoperation may also be useful 
in situations where the number of human beings 
present is limited. For example, a crew of, say, four 
persons on a Mars mission may need various 


medical specialists, but it would not be feasible to 
send them on the mission. One of the characteristics 
of modem medicine is the huge variety of diagnostic 
tests and treatment methods available. Indeed, a 
modem operating room may itself be larger than the 
pressurized volume in a space vehicle of the near 
future. 1 A lone surgeon may need to call on the 
assistance of colleagues on Earth. These Earth- 
bound physicians will have access to huge amounts 
of information from onboard cameras and advanced 
patient monitoring devices. The latter could be worn 
as undergarments 2 or be permanently implanted in 
the bodies of crew members. 1 The huge flow of data 
will have to be managed, and presented to physicians 
in a way that does not cause sensory overload. 
Medical procedures can be developed for situations 
that arise on Earth, and can then be extended to the 
space environment. For example, the costs of 
surgery in remote places such as Antarctica in the 
winter make the development of telesurgery not only 
lifesaving but economical. Any space-based 
construction or colonization will have its share of 
medical/surgical emergencies. It is incumbent on us 
to investigate ways to project life-saving talent into 
orbit via bits instead of mass. 

A major physical constraint on teleoperation 
is the velocity of light. If a human being transmits 
an instruction to a robot on the Moon, and the robot 
transmits a picture of its response to the Earth, the 
operator will not see the response until 2.6 seconds 
after the instruction was sent. This represents the 
round-trip travel time for radio signals traveling 
between the Earth and the Moon. It has been shown 
that the teleoperation of a robot arm using visual 
feedback is feasible under such conditions 3 (see also, 
Ref. 4, Appendix 2, pages 313-326). However, the 
delay time between Earth and Mars will be several 
minutes, rendering such tcleoperation less feasible. 
Furthermore, surgery and other medical procedures 
will require haptic (force) feedback, as well as visual 
feedback, and will be less forgiving of errors. Haptic 
feedback in surgical procedures must usually be 
limited to delays of no more than a few milliseconds, 
although there may be ways of circumventing this in 
some circumstances. The feasibility of teleoperation 
depends on three major factors: 

• the nature of the task itself; 

• the distance over which teleoperation is to take 
place; 
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• the type of feedback (visual, sound, haptic) that 
is available to the teleoperator. 

Both the space and the surgical environment may 
benefit from novel approaches to providing 
feedback. 

Human Factor Limitations on Visual Feedback 

Real-time visual images have been the most 
widely-considered form of feedback in 
teleoperation, simulation, and other computer- 
assisted systems. Computer-generated imagery can 
either augment or replace the unassisted visual field. 
Such systems may consist simply of a color or 
monochromatic monitor, or may involve more 
"immersive" visual devices, such as head- 
mounted/hcads-up displays. The latter may be 
useful in adding computer-generated icons to a "real 
world" visual image. The sophistication of 
computer-generated images is limited by the need to 
render them in real time. Displays are limited in 
terms of scan rate and resolution, and in the case of 
head-mounted displays, comfort. Latency (time 
delay) in image generation and dynamic head 
tracking can cause synthetic imagery to "lag behind" 
real world imagery. For a see-through head- 
mounted display, this will lead to a mismatch 
between the computer-generated image and the "live 
video" of the surroundings. Nausea, or "simulator 
sickness" can result from these lag times, and other 
discrepancies between perceived and expected 
motion cues, such as the absence of vestibular (inner 
ear) feedback (since the eyes may perceive motion 
while the inner ear will not). 5 - 6 - 7 The solution of one 
of these problems may exacerbate others. For 
example, visual displays must operate at speeds 
greater than 30 frames per second in order to 
prevent flickering. However, increasing the frame 
rate will increase computational requirements and 
add to lag time in the generation of images. Even 
without the necessity of combining live and 
computer- generated images, lag times of significant 
fractions of a second will make computer-assisted 
surgery unfeasible. Although the improvement of 
visual systems has been the subject of extensive on- 
going research, the use of audio feedback in virtual 
systems has been relatively unexamined until 
recently. Researchers at Computer-Aided Surgery, 
Inc. (CASI) are currently exploring the use of 
auditory cues as an aid in terrestrial surgery. 8 - 9 - 10 ' 11 * 12 
Researchers at CASI and New York University 
propose to extend these techniques to space-based 
surgery, as well as to other space-based operations, 
such as mining and construction. 

Advantages of Auditory Feedback 

There are several advantages to 
incorporating audio feedback in virtual and 
teleoperation systems. Computational and 
transmission bandwidth requirements are 
substantially smaller than for visual systems. 
Auditory perception is omnidirectional, making it 
possible for the operator to localize sounds coming 


from behind him/her, as well as from behind objects 
in the visual field. The traditional use of sound as a 
warning (e.g., car horns) exploits this property. 
Vertigo and "simulator sickness" do not occur in 
audio systems. Visual systems, while necessary, tend 
to become saturated with information, and human 
beings using them can suffer sensory overload. 
Durand Begault points out that "with an audio 
display, the focus of attention between virtual sound 
sources can be switched at will; vision, on the other 
hand, requires eye or head movement." 13 We 
therefore propose to use audio systems to augment, 
but not to replace, visual systems. 

Sound has been used to reinforce or replace 
other senses, such as in musical user interfaces 
(MUl's) for the blind 14 , warning systems for civil 
aircraft 13 , and audio feedback systems for medical 
equipment. 16 The use of sound to provide positional 
information can be seen as the inverse of the playing 
of variable -pitch musical instruments, such as the 
violin or trombone. With such instruments, which 
lack a mechanism for providing discrete pitches, the 
musician must position his/her hands extremely 
precisely along one axis to control the pitch of the 
sound. Three-axis positioning of the hands is 
necessary to play the theremin, an electronic 
instrument invented in 1928 by Leon Theremin. 17 
The musician controls pitch and amplitude by 
moving his/her hands in space, relative to two 
antennae. In some instruments, the hand must be 
positioned to within fractions of a millimeter. 
Evidence suggests that the inverse process, using 
sound to control position, can provide feedback 
precise enough for surgery, or precision 
manipulation of mechanical components for 
construction purposes. 9 - 10 - ll > 18 

An Experimental Teleoperation System for 
Terrestrial Surgery 

Researchers at Computer Aided Surgery, Inc. 
(CASI) are currently investigating alternative 
approaches for representing information in 
telemanipulation and telementoring applications for 
surgery. The use of programmatically generated 
audio feedback, in a synesthetic paradigm 
(synesthesia from Greek: syn = together + aisthesis; 
defined as a "... physical experience of a cross-modal 
association” 19 ), has proven to be promising both for 
reducing bandwidth requirements, and for creating 
an ergonomic interface. While there is no disputing 
the importance of visual cues in performing intricate 
telemanipulations, we find 12 that it is feasible to use 
audio feedback for reinforcing awareness of 
instrument translation and rotation relative to 3D 
computer models, such as a pre-planned surgical 
tool path. 

We discuss an implementation of an audio 
guidance system for remote surgical manipulation. 
This system consists of an electromagnetic tracking 
device, a workstation-quality computer, and a MIDI 
(Musical Instrument Digital Interface) audio 
synthesizer. For space-based applications, this 



Fig. 1. Audio user interface feedback loop. Without the satellite link, the figure represents an audio guidance 
system for terrestrial surgery. The addition of the satellite link can extend this system to space teleoperation 
for surgery and other purposes. 


system would function as the Earth-bound 
operator’s workstation. A satellite link would 
project the operator's intentions to a remote location 
(Fig. 1). 

In order for the operator’s gestures to have 
any effect upon the synthetic environment, there 
must be a way to track and convert the operator’s 
movements into a machine-readable form. The ideal 
approach would be to use gloves equipped with 
sensors, so that the surgeon could remotely grasp a 
wide range of available objects, but a more practical 
approach is to instrument a surgical tool. The 
surgical tool can serve either as an actual scalpel, or a 
virtual tool, which is used to control an actual scalpel 
(or other device) at a remote location. A tracking 
system employed by CASI consists of two magnetic- 
dipole antennae, one of which functions as a Fixed 
transmitter (the operating table), the other a freely 
moving receiver (a scalpel). Both the transmitter and 
the receiver antennae consist of three mutually 
orthogonal coils with diameters wound as small as 
possible in order that each coil may be regarded as 
an infinitesimal dipole. A pulse of current is sent 
into the transmitter coils. This creates a magnetic 
field which induces currents in the receiver coils. 
These are connected to a signal processing device 
made by Polhemus, Inc. Enough information is 
provided to the device to solve for six unknowns: x, 
y, z, azimuth, elevation, and roll. 20 

The performance of the tracking device has 
an appreciable effect upon the smoothness of 
response of the entire simulator system. If the 
resolution and accuracy are poor, the device will 
prove to be of little use for the ultra-fine 
manipulation required for surgical procedures. If 
the latent period is too long, with the addition of the 
latent periods of all the other components involved 
in the system, the response of the simulation may lag 
too much to be usable in real time. 

The audio feedback signal is generated by 
converting a continuous function of the surgical 
instrument position and orientation with respect to a 
3D model into MIDI device instructions. These are 
instructions for controlling the dedicated audio 
synthesis portion of the hardware. These 


instructions do not convey waveforms, but convey 
control instructions, such as turning a bank of 
oscillators on or off, employing one of a number of 
possible preset waveform tables, to set the pitch of a 
bank of oscillators to one of 0-127 preset 
frequencies (i.e., a chromatic scale, or any 
contiguous series of frequencies roughly defined by 
an interval of an octave/12), or to increase or 
decrease the sum amplitude of a bank of oscillators 
to one of 0-127 possible preset amplitudes. 

The controlling computer generates a stream 
of MIDI instructions using some predesigned 
"sonification" algorithm. This could be as simple as 
generating two continuous tones, one of which is 
fixed as a reference, where the other varies in pitch 
as a function of the error of a cursor in three- 
dimensional space with respect to some pre-planned 
trajectory (with the error calculated as the distance 
perpendicular to the desired trajectoiy). The user 
tries to correct for the error by minimizing the pitch 
interval between the tones. 18 Another scheme might 
be to generate foley effects (impact sounds) when a 
cursor crosses predefined three-dimensional object 
boundaries (such as a scalpel impacting bone). 
Pitch, amplitude, and waveform table parameters 
would change as a function of the impact force, 
cursor orientation (rotation vector), and modeled 
material properties of the surgical tool and tissue. 
The main issue here is one of user interface design; 
to decide which parameters of the interaction are 
important to the operator in performing the task, and 
how to represent this information so that it may be 
most intuitively grasped by the operator. This issue 
can best be addressed by careful examination of the 
entire task scenario, testing and re-engineering of 
candidate algorithms with constant input of potential 
users throughout the entire design process. 

Can Virtual Audio Aid in the Exploration and 
Development of Space? 

The surgical environment, like space, 
presents human beings with an environment that is 
often lacking in familiar reference points. For 
example, endoscopic surgery, which is performed 










through an instrument inserted into the body 
through a small incision, is a technique whose use is 
increasing. It presents both physical and sensory 
challenges to the physician. Surgeons must operate 
with remote-control devices, guided by flat video 
images, without haptic feedback from the organs and 
tissues being manipulated. Researchers at CAS1 are 
investigating ways in which sound cues can aid in 
orienting and guiding a surgeon who is telepresent 
within the patient's body. The authors propose to 
extend these techniques to space construction and 
surgery. 

A major reason for using teleoperation in 
terrestrial surgery is the small size and delicate 
nature of structures in the body; for example, the 
retina. The motions of a surgeon often need to be 
kinematically reduced in both amplitude and force 
by a remote-control apparatus. A telepresence 
surgery system has been devised which can scale 
down the motion of the surgeon's hand by a factor 
of 100, so that 1 cm of hand motion moves the 
surgical apparatus by 0.1 mm. 21 - 22 ’ 23 For space- 
based construction of large structures, the motions of 
the operator may need to be kinematically amplified 
by this amount or more. A similar situation exists 
for sensory feedback. Microsurgery requires not 
only remote manipulation, but remote stereoscopic 
vision. Space-based construction can be seen as 
"macrosurgery," requiring stereo cameras mounted 
on a long baseline to extend the operator's depth 
perception. 

The lag time caused by the processing of 
visual images will be exacerbated by speed-of-light 
delays if the operator is a significant distance from 
the construction site (say. Earth to geostationary 
orbit, or even Earth to Moon). Here, sound can 
function in a variety of ways. By reducing the 
bandwidth requirements, audio feedback can make 
teleoperation more feasible over a given distance, 
and can extend the distance over which teleoperation 
is practical. The operator may benefit from an 
intuitive sensory indication of the time delay. 
Multiple parallel independent channels of situational 
awareness information can be encoded in parallel 
musical "voices." Distance can be indicated by 
amplitude or pitch, and speed can be indicated by 
change in pitch (Doppler shift). Teleoperating from 
the Earth to the Moon can be performed in a 
manner similar to that of athletes, who must throw 
and catch a ball whose time of flight may be several 
seconds. A teleoperator on Earth may therefore be 
able to "throw” his/her intentions to an apparatus on 
the Moon; he/she can then "catch" the result as visual 
feedback is "thrown" back to Earth at the speed of 
light. This visual feedback can be enhanced with 
sound cues, so that the operator need not guess when 
the system is ready for another command. This may 
be especially important for teleoperation to (or 
from) a moving spacecraft. Over the course of an 
operation (surgical or otherwise) which lasts for 
several hours, a spacecraft moving at Earth-escape 
velocity may travel a sufficient fraction of a light- 


second. A teleoperator will have to continually 
readjust the pace of his/her reactions. Sound cues 
may be able to aid in this adjustment. 

Many teleoperated tasks, particularly 
surgery, can benefit from haptic feedback. This 
must be limited to delays of no more than a few 
milliseconds. At first, it may seem that tasks 
involving haptic feedback are feasible over short 
distances only; say, between the surface of the Earth 
and low Earth orbit. However, it may be possible to 
increase the machine intelligence of the 
teleoperator's work station. It can be programmed to 
provide pre-determined responses that match the 
likely feedback expected from the remote end. As 
the true response is received, the work station can be 
programmed to leant from experience, and correct 
for discrepancies between expected and actual 
responses. Similarly, actions that require an 
immediate response can benefit from increased 
intelligence at the remote location of the 
"teleoperatee." We define the latter term as that 
which is affected by the actions of the teleoperator; it 
can be a machine(s), a human being(s), or a 
combination of both. The data stream from the 
teleoperatee may be images from an endoscopic 
camera, • telemetry from patient monitoring 
equipment, or positional information from a robot 
arm used in space construction. The teleoperator 
will need to have this information presented in a 
manner that maintains his/her orientation relative to 
the task at hand, without causing sensory overload. 
Dividing the intelligence between the local and 
remote ends is similar to the way that the human 
body overcomes the limitations of the time-of-flight 
of nervous signals by the use of nerve ganglia and 
reflex loops to reduce the informational load of the 
distant brain (in this case, the teleoperator). Finally, 
a teleoperator can act as an avatar, or disembodied 
advisor, seeing, hearing, and feeling everything a 
teleoperatee experiences, except delayed in time. 
The avatar/operator can impart, by suggestion, 
gesture, and illustration, the essential information 
required to accomplish a task even when lagging 
many minutes behind the teleoperatee. 

As the space infrastructure develops, this 
concept of "ganglia" or local nerve centers, can be 
extended further. For example, teleoperated robots 
can be deployed at various locations on the lunar 
surface for the purpose of geological surveys and 
prospecting. 24 The operator can be stationed on the 
Earth. A portion of the remote intelligence can be 
installed on a space platform at the LI Libration 
Point (see Fig. 2). This can keep the robots small 
and light, while keeping time delays to a minimum. 
Furthermore, upgrades and repairs to the system can 
be performed at one location, rather than recalling 
the robots or deploying new ones. Fig. 2 shows a 
schematic diagram of the Earth-Moon system. The 
various locations in space may take on the role of 
nerve centers. The exact deployment of human 
beings and equipment will depend on the specific 
economic path that space development takes. 


However, if teleoperation is a major factor, 
minimizing delays due to the speed of light will be 
important. It should be noted that the round-trip 
speed of light delays indicated in Fig. 2 are 
idealizations. In practice, some of these will be 
longer. In particular, transmitting information 
between a point on the Earth's surface to any orbit 
around the Earth will involve an indirect route 
involving numerous repeaters and stations on Earth 
and in orbit, in order to overcome the constraint of 
the Earth’s rotation. A similar situation exists for 
non-stationary orbits. Thus, teleoperating from L5 
to the Moon may be more feasible than 
teleoperating from the Earth to the Moon. However, 
it does not seem likely that a human habitat will be 
established at L5 just for the purpose of 


teleoperation. If such a habitat is built for, say, 
materials processing, however, it is a straightforward 
matter to station teleoperators there as well, perhaps 
with the LI point serving as a local nerve center. (It 
should be noted that due to perturbations from the 
Sun's gravity, L4 and L5 are not stable points. 
However, there are stable regions which move in 
orbits about them [Ref. 4, page 127], so this should 
not significantly affect the teleoperation strategy. 
An object at LI or L2 will be unstable in the 
direction of the Earth-Moon axis, but its position can 
be maintained by the use of thrusters.) The use of 
teleoperation can thus grow along with the rest of the 
space infrastructure. 



Fig. 2. Schematic illustration of the Earth-Moon system (not to scale). Prograde orbits arc shown for an 
observer in the north celestial hemisphere. LI, L2, etc. represent the five Lagrange Iibration points. An 
altitude of 400 km was used for low Earth orbit. Times indicate round-trip velocity of light delays 
(idealizations which do not include further delays needed to relay signals to/from a rotating Earth or non- 
stationary orbits). Time delays between L4 and the other points shown are the same as those for L5. The 
Lagrange points can be used as nerve centers for a space tcleoperation system. 


Conclusions 

Telepresence and virtual reality are among 
today’s fastest-growing technologies. The immense 
throughput of data in teleoperation and virtual 
systems will likely increase as the technology 
progresses. The bottleneck in dealing with this data 
stream is, of course, the human brain. Rather than 
overloading the brain, teleoperation should make 
information management easier. The use of sound 
cues shows great promise in this area. Much of our 
effort to date has involved orienting the operator in 
the "inner space" of the human body. A logical 
extension of this work will be to adapt it for use in 
outer space. A near-term application may be the 
addition of sound effects to the virtual environment 
to substitute for the lack of sound in the vacuum of 
space. Adapting to the high frontier will require that 
human beings use their age-old capabilities in new 
ways. For example, it is common for astronauts to 
develop "space sickness." This is the nausea that 
results from conflicting sensory cues. The 
astronaut’s eyes may provide a sense of up/down as 
a direction away or toward the feet. However, the 
vestibular sense from the inner ear provides no 
up/down sense without a gravitational gradient. We 
intend to investigate the use of sound cues to serve as 
a replacement for gravitational gradient cues in 
helping human beings orient themselves in space 
and avoid nausea. Appropriately, one function of 
the ear (hearing) may substitute for the other (the 
vestibular sense). Again, the precise auditory input 
used, the exact combination of human beings and 
machines that will prove to be optimal, and other 
specifics of the developing space infrastructure are 
difficult to predict. But we must prepare for life 
beyond the Earth by developing the means to 
effectively project our capabilities into a new 
environment. 
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